PatchUp : a feature-space block-level regularization technique for convolutional neural networks

Les modèles d’apprentissage profond à large capacité ont souvent tendance à présenter de hauts écarts de généralisation lorsqu’ils sont entrainés avec une quantité limitée de données étiquetées. Dans ce cas, des réseaux de neurones très profonds et larges auront tendance à mémoriser les échantillons de données et donc ils risquent d’être vulnérables lors d’un léger décalage dans la distribution des données au moment de tester. Ce problème produit une généralisation pauvre lors de changements dans la répartition des données au moment du test. Pour surmonter ce problème, certaines méthodes basées sur la dépendance et l’indépendance de données ont été proposées. Une récente classe de méthodes efficaces pour aborder ce problème utilise plusieurs manières de contruire un nouvel échantillon d’entrainement, en mixant une paire (ou plusieurs) échantillons d’entrainement. Dans cette thèse, nous introduisons PatchUp, une régularisation de l’espace des caractéristiques au niveau des blocs dépendant des données qui opère dans l’espace caché en masquant des blocs contigus parmi les caractéristiques mappées, sélectionnés parmi une paire aléatoire d’échantillons, puis en mixant (Soft PatchUp) ou en échangeant (Hard PatchUp) les blocs contigus sélectionnés. Notre méthode de régularisation n’ajoute pas de surcharge de calcul significative au CNN pendant l’entrainement du modèle. Notre approche améliore la robustesse des modèles CNN face au problème d’intrusion du collecteur qui pourrait apparaitre dans d’autres approches de mixage telles que Mixup et CutMix. De plus, vu que nous mixons des blocs contigus de caractéristiques dans l’espace caché, qui a plus de dimensions que l’espace d’entrée, nous obtenons des échantillons plus diversifiés pour entrainer vers différentes dimensions. Nos expériences sur les ensembles de données CIFAR-10, CIFAR-100, SVHN et Tiny-ImageNet avec des architectures ResNet telles que PreActResnet18, PreActResnet34, WideResnet-28-10, ResNet101 et ResNet152 montrent que PatchUp dépasse ou égalise les performances de méthodes de régularisation pour CNN considérée comme état de l’art actuel. Nous montrons aussi que PatchUp peut fournir une meilleure généralisation pour des transformations affines d’échantillons et est plus robuste face à des attaques d’exemples contradictoires. PatchUp aide aussi les modèles CNN à produire une plus grande variété de caractéristiques dans les blocs résiduels en comparaison avec les méthodes de pointe de régularisation pour CNN telles que Mixup, Cutout, CutMix, ManifoldMixup et Puzzle Mix. Mots clés: Apprentissage en profondeur, Réseau Neuronal Convolutif, Généralisation,Régularisation, Techniques de régularisation dépendantes et indépendantes des données, Robustesse aux attaques adverses.Large capacity deep learning models are often prone to a high generalization gap when trained with a limited amount of labeled training data. And, in this case, very deep and wide networks have a tendency to memorize the samples, and therefore they might be vulnerable under a slight distribution shift at testing time. This problem yields poor generalization for data outside of the training data distribution. To overcome this issue some data-dependent and data-independent methods have been proposed. A recent class of successful methods to address this problem uses various ways to construct a new training sample by mixing a pair (or more) of training samples. In this thesis, we introduce PatchUp, a feature-space block-level data-dependent regularization that operates in the hidden space by masking out contiguous blocks of the feature map of a random pair of samples, and then either mixes (Soft PatchUp) or swaps (Hard PatchUp) these selected contiguous blocks. Our regularization method does not incur significant computational overhead for CNNs during training. Our approach improves the robustness of CNN models against the manifold intrusion problem that may occur in other state-of-the-art mixing approaches like Mixup and CutMix. Moreover, since we are mixing the contiguous block of features in the hidden space, which has more dimensions than the input space, we obtain more diverse samples for training towards different dimensions. Our experiments on CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet datasets using ResNet architectures including PreActResnet18, PreActResnet34, WideResnet-28-10, ResNet101, and ResNet152 models show that PatchUp improves upon, or equals, the performance of current state-of-the-art regularizers for CNNs. We also show that PatchUp can provide a better generalization to affine transformations of samples and is more robust against adversarial attacks. PatchUp also helps a CNN model to produce a wider variety of features in the residual blocks compared to other state-of-the-art regularization methods for CNNs such as Mixup, Cutout, CutMix, ManifoldMixup, and Puzzle Mix. Key words: Deep Learning, Convolutional Neural Network, Generalization, Regular-ization, Data-dependent and Data-independent Regularization Techniques, Robustness to Adversarial Attacks

An evolutionary polynomial regression (EPR) model for prediction of H2S induced corrosion in concrete sewer pipes

The sulphuric acid is a known growing threat to concrete sewer pipes. Acid production is dictated by rapid urbanisation, increased use of hot water and discharge of toxic metals and sulphate containing detergents into the wastewater. Concrete sewer pipe corrosion due to sulphuric attack is known to be the main contributory factor of pipe degradation. Very little tools are available to accurately predict the corrosion rate and most importantly the remaining safe life of the asset. This paper proposes a new robust model to predict the sewer pipe corrosion rate due to sulphuric acid. The model makes use of a powerful Evolutionary Polynomial Regression method that provides a new methodology of hybrid data-mining. The results obtained by the model which was validated in the field indicates that the proposed hybrid methodology can accurately predict the corrosion rate in concrete sewer pipe’s given that the pipe installation conditions as well as in-pipe sewage conditions are known

Advanced numerical and analytical methods for assessing concrete sewers and their remaining service life

Pipelines are extensively used engineering structures which convey fluid from one place to another. Most of the time, pipelines are placed underground and are encumbered by soil weight and traffic loads. Corrosion of pipe material is the most common form of pipeline deterioration and should be considered in both the strength and serviceability analysis of pipes. The study in this research focuses on concrete pipes in sewage systems (concrete sewers). This research firstly investigates how to involve the effect of corrosion as a time dependent process of deterioration in the structural and failure analysis of this type of pipe. Then three probabilistic time dependent reliability analysis methods including the first passage probability theory, the gamma distributed degradation model and the Monte Carlo simulation technique are discussed and developed.Sensitivity analysis indexes which can be used to identify the most important parameters that affect pipe failure are also discussed. The reliability analysis methods developed in this paper contribute as rational tools for decision makers with regard to the strengthening and rehabilitation of existing pipelines. The results can be used to obtain a cost-effective strategy for the management of the sewer system

PatchUp: A Regularization Technique for Convolutional Neural Networks

Large capacity deep learning models are often prone to a high generalization gap when trained with a limited amount of labeled training data. A recent class of methods to address this problem uses various ways to construct a new training sample by mixing a pair (or more) of training samples. We propose PatchUp, a hidden state block-level regularization technique for Convolutional Neural Networks (CNNs), that is applied on selected contiguous blocks of feature maps from a random pair of samples. Our approach improves the robustness of CNN models against the manifold intrusion problem that may occur in other state-of-the-art mixing approaches like Mixup and CutMix. Moreover, since we are mixing the contiguous block of features in the hidden space, which has more dimensions than the input space, we obtain more diverse samples for training towards different dimensions. Our experiments on CIFAR-10, CIFAR-100, and SVHN datasets with PreactResnet18, PreactResnet34, and WideResnet-28-10 models show that PatchUp improves upon, or equals, the performance of current state-of-the-art regularizers for CNNs. We also show that PatchUp can provide better generalization to affine transformations of samples and is more robust against adversarial attacks

An Introduction to Lifelong Supervised Learning

This primer is an attempt to provide a detailed summary of the different facets of lifelong learning. We start with Chapter 2 which provides a high-level overview of lifelong learning systems. In this chapter, we discuss prominent scenarios in lifelong learning (Section 2.4), provide 8 Introduction a high-level organization of different lifelong learning approaches (Section 2.5), enumerate the desiderata for an ideal lifelong learning system (Section 2.6), discuss how lifelong learning is related to other learning paradigms (Section 2.7), describe common metrics used to evaluate lifelong learning systems (Section 2.8). This chapter is more useful for readers who are new to lifelong learning and want to get introduced to the field without focusing on specific approaches or benchmarks. The remaining chapters focus on specific aspects (either learning algorithms or benchmarks) and are more useful for readers who are looking for specific approaches or benchmarks. Chapter 3 focuses on regularization-based approaches that do not assume access to any data from previous tasks. Chapter 4 discusses memory-based approaches that typically use a replay buffer or an episodic memory to save subset of data across different tasks. Chapter 5 focuses on different architecture families (and their instantiations) that have been proposed for training lifelong learning systems. Following these different classes of learning algorithms, we discuss the commonly used evaluation benchmarks and metrics for lifelong learning (Chapter 6) and wrap up with a discussion of future challenges and important research directions in Chapter 7.Comment: Lifelong Learning Prime

Prediction of sulphide build-up in filled sewer pipes

Millions of dollars are being spent worldwide on the repair and maintenance of sewer networks and wastewater treatment plants. The production and emission of hydrogen sulphide has been identified as a major cause of corrosion and odour problems in sewer networks. Accurate prediction of sulphide build-up in a sewer system helps engineers and asset managers to appropriately formulate strategies for optimal sewer management and reliability analysis. This paper presents a novel methodology to model and predict the sulphide build-up for steady state condition in filled sewer pipes. The proposed model is developed using a novel data-driven technique called evolutionary polynomial regression (EPR) and it involves the most effective parameters in the sulphide build-up problem. EPR is a hybrid technique, combining genetic algorithm and least square. It is shown that the proposed model can provide a better prediction for the sulphide build-up as compared with conventional models

Biosynthesis and recovery of selenium nanoparticles and the effects on matrix metalloproteinase-2 expression

Today, green synthesis of nanoparticles is attracting increasing attention. In the present study, the Bacillus sp. MSh-1 was isolated from the Caspian Sea (located in the northern part of Iran) and identified by various identification tests and 16S ribosomal DNA analysis. The reduction time course study of selenium ion (Se4+) reduction by using this test strain was performed in a liquid culture broth. Then, the intracellular NPs (nanoparticles) were released by the liquid nitrogen disruption method and thoroughly purified using an n-octyl alcohol water extraction system. Characterization of the separated NPs on features such as particle shape, size and purity was carried out with different devices. The energy dispersive X-ray and X-ray diffraction patterns showed that the purified NPs consisted of only selenium and are amorphous respectively. In addition, the transmission electron micrograph showed that the separated NPs were spherical and 80–220 nm in size. Furthermore, the cytotoxicity effect of these extracted biogenic selenium (Se) NPs on the fibrosarcoma cell line (HT-1080) proliferation and the inhibitory effect of the Se NPs on MMP-2 (matrix metalloproteinase-2) expression were studied using the MTT [3-(4,5-dimethylthiazol- 2-yl)-2,5-diphenyl-2H-tetrazolium bromide] assay and gelatin zymography. Biogenic Se NPs showed a moderately inhibitory effect on MMP-2 expression

Photocatalytic decolorization of bromothymol blue using biogenic selenium nanoparticles synthesized by terrestrial actinomycete Streptomyces griseobrunneus strain FSHH12

The aim of the present study was to isolate and identify a terrestrial actinomycete bacterial strain capable to produce selenium nanoparticles (Se NPs) followed by purification of the biogenic Se NPs and evaluation of their photocatalytic degradation compared to selenium dioxide. Among 30 actinomycete bacterial strains obtained from environmental soil samples, one isolate (identified as Streptomyces griseobrunneus strain FSHH12 based on the 16S rDNA gene sequence analysis) was selected and used for production of Se NPs. The biologically synthesized Se NPs was consequently purified by an organic–aqueous partitioning system and characterized using scanning electron microscopy, transmission electron microscopy, energy dispersive X-ray, UV–visible spectroscopy, Fourier transform infrared spectroscopy, and X-ray diffraction spectroscopy. The obtained results of photocatalytic degradation of bromothymol blue using the purified Se NPs (64 μg/mL) revealed 62.3% of dye removal under UV illumination (15 W) after 60 min incubation of dye solution

Biosynthesis and recovery of rod-shaped tellurium nanoparticles and their bactericidal activities

In this study, a tellurium-transforming Bacillus sp. BZ was isolated from the Caspian Sea in northern Iran. The isolate was identified by various tests and 16S rDNA analysis, and then used to prepare elemental tellurium nanoparticles. The isolate was subsequently used for the intracellular biosynthesis of elemental tellurium nanoparticles. The biogenic nanoparticles were released by liquid nitrogen and purified by an n-octyl alcohol water extraction system. The shape, size, and composition of the extracted nanoparticles were characterized. The transmission electron micrograph showed rod-shaped nanoparticles with dimensions of about 20 nm � 180 nm. The energy dispersive X-ray and X-ray diffraction spectra respectively demonstrated that the extracted nanoparticles consisted of only tellurium and have a hexagonal crystal structure. This is the first study to demonstrate a biological method for synthesizing rod-shaped elemental tellurium by a Bacillus sp., its extraction and its antibacterial activity against different clinical isolates

Green synthesis of gold nanoparticles by the marine microalga Tetraselmis suecica

The application of green-synthesis principles is one of the most impressive research fields for the production of nanoparticles. Different kinds of biological systems have been used for this purpose. In the present study, AuNPs (gold nanoparticles) were prepared within a short time period using a fresh cell extract of the marine microalga Tetraselmis suecica as a reducing agent of HAuCl4 (chloroauric acid) solution. The UV–visible spectrum of the aqueous medium containing AuNPs indicated a peak at 530 nm, corresponding to the surface plasmon absorbance of AuNPs. The X-ray diffraction pattern also showed a Bragg reflection related to AuNPs. Fourier-transform infrared spectroscopy was performed for analysis of surface functional groups of AuNPs. Transmission electron microscopy and particle-size-distribution patterns determined by the laser-light-scattering method confirmed the formation of well-dispersed AuNPs. The most frequent size of particles was 79 nm